Leveraging derivative virtual machine introspection methods for security applications

Virtual machine introspection (VMI) describes the method of monitoring, analyzing, and manipulating the state of a virtual machine from the hypervisor level. This lends itself to many security applications, though they all share a single fundamental challenge: One must address the fact that the hypervisor has no semantic knowledge about what the system state means (e. g., where key data structures are). Traditionally, this semantic knowledge is simply delivered to the hypervisor in the form of debugging symbols, symbol tables, etc. If such semantic information does not exist, it must be acquired through other, often tedious, means such as reverse engineering or “trial and error”. On the other hand, it is possible to derive information about the guest OS by considering hardware features and their specifications. This derivative method is possible without any delivered semantic information about the guest OS and has several additional advantages including guest OS portability and resistance to evasion techniques. The main contribution of this research is an examination of derivative VMI methods and their strengths. As there is little prior work formally exploring the potential of derivative VMI methods, we inspect Intel’s IA-32 and IA-32e architectures and investigate their potential for derivative VMI. Through this inspection, we discover and present several portions of the hardware specifications which are conducive to derivative VMI. This culminates in the introduction of a novel derivative method for collecting system calls from the hypervisor. This method is completely guest OS agnostic and has been tested on a variety of guest OSs. In addition, our method cannot be evaded from within the guest. Furthermore, we show that our method keeps the collection overhead to a minimum by comparing its performance to a similar system. We extend this work by presenting a novel approach to malware detection that makes use of a support vector machine (SVM) to classify system call traces. In contrast to other methods that use system call traces for malware detection, our approach employs a string kernel to make better use of the sequential information inherent in a system call trace. By classifying system call traces in small sections and keeping a moving average over the probability estimates produced by the SVM, our approach is capable of detecting malicious behavior online and achieves a very high accuracy.